Goto

Collaborating Authors

 Iqaluit


COPU: Conformal Prediction for Uncertainty Quantification in Natural Language Generation

Wang, Sean, Jiang, Yicheng, Tang, Yuxin, Cheng, Lu, Chen, Hanjie

arXiv.org Artificial Intelligence

Uncertainty Quantification (UQ) for Natural Language Generation (NLG) is crucial for assessing the performance of Large Language Models (LLMs), as it reveals confidence in predictions, identifies failure modes, and gauges output reliability. Conformal Prediction (CP), a model-agnostic method that generates prediction sets with a specified error rate, has been adopted for UQ in classification tasks, where the size of the prediction set indicates the model's uncertainty. However, when adapting CP to NLG, the sampling-based method for generating candidate outputs cannot guarantee the inclusion of the ground truth, limiting its applicability across a wide range of error rates. To address this, we propose \ourmethod, a method that explicitly adds the ground truth to the candidate outputs and uses logit scores to measure nonconformity. Our experiments with six LLMs on four NLG tasks show that \ourmethod outperforms baseline methods in calibrating error rates and empirical cover rates, offering accurate UQ across a wide range of user-specified error rates.


Elaborative Subtopic Query Reformulation for Broad and Indirect Queries in Travel Destination Recommendation

Wen, Qianfeng, Liu, Yifan, Zhang, Joshua, Saad, George, Korikov, Anton, Sambale, Yury, Sanner, Scott

arXiv.org Artificial Intelligence

In Query-driven Travel Recommender Systems (RSs), it is crucial to understand the user intent behind challenging natural language (NL) destination queries such as the broadly worded "youth-friendly activities" or the indirect description "a high school graduation trip". Such queries are challenging due to the wide scope and subtlety of potential user intents that confound the ability of retrieval methods to infer relevant destinations from available textual descriptions such as WikiVoyage. While query reformulation (QR) has proven effective in enhancing retrieval by addressing user intent, existing QR methods tend to focus only on expanding the range of potentially matching query subtopics (breadth) or elaborating on the potential meaning of a query (depth), but not both. In this paper, we introduce Elaborative Subtopic Query Reformulation (EQR), a large language model-based QR method that combines both breadth and depth by generating potential query subtopics with information-rich elaborations. We also release TravelDest, a novel dataset for query-driven travel destination RSs. Experiments on TravelDest show that EQR achieves significant improvements in recall and precision over existing state-of-the-art QR methods.


Scalable mixed-domain Gaussian process modeling and model reduction for longitudinal data

Timonen, Juho, Lähdesmäki, Harri

arXiv.org Artificial Intelligence

Gaussian process (GP) models that combine both categorical and continuous input variables have found use in longitudinal data analysis of and computer experiments. However, standard inference for these models has the typical cubic scaling, and common scalable approximation schemes for GPs cannot be applied since the covariance function is non-continuous. In this work, we derive a basis function approximation scheme for mixed-domain covariance functions, which scales linearly with respect to the number of observations and total number of basis functions. The proposed approach is naturally applicable to also Bayesian GP regression with discrete observation models. We demonstrate the scalability of the approach and compare model reduction techniques for additive GP models in a longitudinal data context. We confirm that we can approximate the exact GP model accurately in a fraction of the runtime compared to fitting the corresponding exact model. In addition, we demonstrate a scalable model reduction workflow for obtaining smaller and more interpretable models when dealing with a large number of candidate predictors.


Sparse Variational Contaminated Noise Gaussian Process Regression with Applications in Geomagnetic Perturbations Forecasting

Iong, Daniel, McAnear, Matthew, Qu, Yuezhou, Zou, Shasha, Toth, Gabor, Chen, Yang

arXiv.org Artificial Intelligence

GPR models can also incorporate prior knowledge through selecting an appropriate kernel function. GPR commonly assumes a homoscedastic Gaussian distribution for observation noise because this yields an analytical form for the posterior predictive prediction. However, Bayesian inference based on Gaussian noise distributions is known to be sensitive to outliers which are defined as observations that strongly deviate from model assumptions. In regression, outliers can arise from relevant inputs being absent from the model, measurement error, and other unknown sources. These outliers are associated with unconsidered sources of variation that affect the target variable sporadically. In this case, the observation model is unable to distinguish between random noise and systematic effects not captured by the model. In the context of GPR under Gaussian noise, outliers can heavily influence the posterior predictive distribution, resulting in a biased estimate of the mean function and overly confident prediction intervals. Therefore, robust observation models are desired in the presence of potential outliers.


A gray-box model for a probabilistic estimate of regional ground magnetic perturbations: Enhancing the NOAA operational Geospace model with machine learning

Camporeale, Enrico, Cash, M. D., Singer, H. J., Balch, C. C., Huang, Z., Toth, G.

arXiv.org Machine Learning

We present a novel algorithm that predicts the probability that time derivative of the horizontal component of the ground magnetic field $dB/dt$ exceeds a specified threshold at a given location. This quantity provides important information that is physically relevant to Geomagnetically Induced Currents (GIC), which are electric currents induced by sudden changes of the Earth's magnetic field due to Space Weather events. The model follows a 'gray-box' approach by combining the output of a physics-based model with a machine learning approach. Specifically, we use the University of Michigan's Geospace model, that is operational at the NOAA Space Weather Prediction Center, with a boosted ensemble of classification trees. We discuss in detail the issue of combining a large dataset of ground-based measurements ($\sim$ 20 years) with a limited set of simulation runs ($\sim$ 2 years) by developing a surrogate model for the years in which simulation runs are not available. We also discuss the problem of re-calibrating the output of the decision tree to obtain reliable probabilities. The performance of the model is assessed by typical metrics for probabilistic forecasts: Probability of Detection and False Detection, True Skill Score, Heidke Skill Score, and Receiver Operating Characteristic curve.


Order of Canada marks 50 years of honouring Canadian contributions - The Globe and Mail

#artificialintelligence

The Order of Canada marks its 50th anniversary this year with 99 new appointments on its Canada Day honours list, including renowned figures from the fields of law, government, entertainment and sport, as well as Canadians whose contributions are less widely known. The list includes soccer star Christine Sinclair, television host Alex Trebek, actor Catherine O'Hara and Globe and Mail editorial cartoonist Brian Gable. Three people were named to the highest rank, Companion of the Order of Canada: former Supreme Court Justice Marshall Rothstein, National Arts Centre president Peter Herrndorf and The Prince of Wales. Nineeteen people were named Officers of the Order of Canada, including former spymaster Richard Fadden, hockey player Mark Messier and actor Michael Myers. There were 77 people named as members of the Order, including opera singer Tracy Dahl, historian Bill Waiser, public health nurse Cathy Crowe and Indigenous leader Terrance Paul.